Three-dimensional image processing apparatus and three-dimensional image processing method

ABSTRACT

In a three-dimensional image capturing apparatus (three-dimensional image processing apparatus), a depth obtainment unit obtains L depth information and R depth information from a three-dimensional image, and an image correction unit executes a smoothing process on an end part region of the subject based on the L depth information and the R depth information. As a result, the three-dimensional image processed by the three-dimensional image processing apparatus is a high-quality three-dimensional image that correctly expresses a sense of three-dimensionality and thickness in the subject and that has a low sense of a cardboard cutout effect.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT Application No.PCT/JP2011/006390 filed Nov. 16, 2011 which claims priority to JapaneseApplication Number 2011-009049 filed Jan. 19, 2011. The entiredisclosures of PCT Application No. PCT/JP2011/006390 and JapaneseApplication Number 2011-009049 are hereby incorporated herein byreference.

BACKGROUND

1. Field of the Invention

The present invention relates to techniques for increasing the qualityof a three-dimensional image (a three-dimensional stereoscopic image),and relates to techniques that can be applied in a wide range of devicesthat handle three-dimensional images (three-dimensional video), such ascameras (image capturing apparatuses) that capture three-dimensionalimages, display apparatuses that display three-dimensional images(three-dimensional video), image processing apparatuses that processthree-dimensional images (three-dimensional video), and so on.

2. Description of the Related Art

Three-dimensional image capturing apparatuses that capturethree-dimensional images in a state where binocular disparity is present(that is, capture a left eye image and a right eye image) are known;such apparatuses make it possible to reproduce a three-dimensional imagein a display apparatus (called a “three-dimensional display apparatus”hereinafter) capable of projecting the three-dimensional image (the lefteye image and the right eye image) independently for the left and righteyes, respectively.

In three-dimensional image capturing, a three-dimensional image (a lefteye image and a right eye image) obtained in a state in which a highlevel of disparity is present in a far scene (a subject in the farscene) or a near scene (a subject in the near scene) will result in animage that exceeds the fusion limit for three-dimensional viewing by aperson and is thus difficult to appear as three-dimensional, or an imagethat produces a sense of fatigue in people who are viewing thethree-dimensional image (a tiring image). In order to avoid generatingsuch a poor three-dimensional image, there are techniques that obtainfavorable three-dimensional images by performing disparity adjustment,stereo base adjustment (called “SB adjustment” hereinafter), and so on,and such techniques are widely used in professional three-dimensionalimage capturing for movies and the like.

Disparity adjustment is a technique used primarily in the case where afar scene (a subject in the far scene) exceeds the fusion limit, andadjusting the disparity so that the distance to the far scene iscompressed in a nonlinear manner brings the far scene (the subject inthe far scene) that was difficult to see three-dimensionally nearer,making it possible to obtain a three-dimensional image that is easy toperceive in three dimensions (a three-dimensional image that can easilybe seen in three dimensions).

On the other hand, stereo base adjustment reduces the space between twocameras (a camera for capturing a left eye image and a camera forcapturing a right eye image) (that is, reduces the stereo base(interaxial distance)), making it possible to reduce the dynamic rangeof the disparity. For this reason, capturing a three-dimensional imageafter performing the stereo base adjustment described above makes itpossible to obtain a three-dimensional image in which the entire scene,from the far scene (a subject in the far scene) to the near scene (asubject in the near scene), is within a fusional area.

In addition, even in the case where the three-dimensional image isdisplayed in a display apparatus at a small size, the disparity of thethree-dimensional image (that is, between the left eye image and theright eye image) is reduced, and thus the far scene is compressed.Accordingly, in this case, the three-dimensional image displayed in thesmall-size display apparatus is a three-dimensional image that is easyto view.

Employing the stated image capturing techniques (disparity adjustment,stereo base adjustment) in three-dimensional image capturing makes itpossible to capture a three-dimensional image that is sufficiently easyto view (that is, a three-dimensional image that is easily perceptiblein three dimensions) when displaying the image in three dimensions in apredetermined display environment (for example, see Japanese PatentH8-9421A).

However, in the aforementioned conventional technique, athree-dimensional image that is easy to view (that is, athree-dimensional image that is easily perceptible in three dimensions)is obtained by taking the fusion limit for three-dimensional viewinginto consideration and reducing the desired disparity (that is, byreducing the disparity from its original value so that the subject thatis the target of the three-dimensional image capturing falls within thefusional area for three-dimensional viewing), and is therefore notdesirable from the standpoint of obtaining a natural sense ofthree-dimensionality and depth in the three-dimensional image.Accordingly, three-dimensional images using the aforementionedconventional techniques (techniques employing disparity adjustment andstereo base adjustment) have a problem in terms of the quality of thethree-dimensional images.

Techniques employing disparity adjustment can obtain three-dimensionalimages that are easy to view (that is, that are easily perceptible inthree dimensions), but because the distance to the far scene iscompressed in a nonlinear manner, a phenomenon in which the far sceneappears as a flat plane (that is, a phenomenon in which a sense ofthickness in subjects in the far scene is reduced and the subjectsappear as flattened three-dimensional images) occurs inthree-dimensional images on which disparity adjustment has beenperformed.

Meanwhile, techniques employing SB adjustment have an overall reducedsense of depth in the three-dimensional images that are obtained (thatis, the distance from the closest point to the farthest point isreduced), and thus a phenomenon in which the sense ofthree-dimensionality of individual subjects is reduced occurs.

Accordingly, the three-dimensional images obtained using any of theaforementioned conventional techniques tend to be images having a poorsense of three-dimensionality and depth, and thus have poor quality.

In addition, there are cases where what is known as a “cardboard cutouteffect” occurs due to the compression/reduction in the sense ofthree-dimensionality arising in the case where the aforementionedconventional techniques are used.

The “cardboard cutout effect” is a phenomenon in which, in athree-dimensional image, the thickness of, for example, a primarysubject such as a person in the near scene is reduced, and the subjectresembles a flat picture drawn on a board.

If this cardboard cutout effect occurs in a primary subject, which is ofhigh importance, there will be an extreme drop in the quality of thethree-dimensional image.

However, the cardboard cutout effect does not occur only due to thecompression/reduction in the sense of three-dimensionality arising inthree-dimensional images due to the disparity adjustment as in theaforementioned conventional techniques. Depending on the image capturingconditions (image capturing state), the cardboard cutout effect canoccur even in ideal, undistorted three-dimensional image capturing(image capturing that captures three-dimensional images with nocompression/reduction in the sense of three-dimensionality).

Accordingly, the cardboard cutout effect is a visual phenomenon, and allof the causes of the cardboard cutout effect have not necessarily beenclarified. However, regardless of the cause of the cardboard cutouteffect occurring, the effect always reduces the quality ofthree-dimensional images.

Having been achieved in light of the aforementioned problems, it is anobject of the present invention to realize a three-dimensional imageprocessing apparatus, a three-dimensional image processing method, and aprogram that restore a sense of three-dimensionality and thickness to asubject and obtain a high-quality three-dimensional image with a lowsense of a cardboard cutout effect, regardless of the causes of thecardboard cutout effect.

SUMMARY

A first aspect of the invention is a three-dimensional image processingapparatus that performs an image correction process on a left eye imageand a right eye image contained in a three-dimensional image obtained bya dual-lens technique or a multiple-viewpoint technique, and includes anend part region detection unit and an edge correction unit.

The end part region detection unit detects, from the left eye image andthe right eye image, one or both of a region including an edge on a leftside of a subject contained in the left eye image and the right eyeimage and a region including an edge on a right side of the subjectcontained in the left eye image and the right eye image, as an end partregion.

The edge correction unit executes a smoothing process on a region in atleast one end part region of the subject detected by the end part regiondetection unit.

According to this three-dimensional image processing apparatus, the endpart region of the subject is detected in the left eye image and/or theright eye image, and a smoothing process is executed on at least one endof the detected end part region of the subject. As a result, accordingto this three-dimensional image processing apparatus, it is possible tosuitably prevent a drop in the quality of a sense ofthree-dimensionality/sense of depth (for example, a drop in qualitycaused by a cardboard cutout effect, a phenomenon in which frosted glassappears to follow the subject, and so on) arising due to the end partregion of the subject inappropriately undergoing disparity matching inthe left eye image and the right eye image.

Accordingly, a three-dimensional image obtained by the three-dimensionalimage processing apparatus is a high-quality three-dimensional image inwhich the occurrence of the cardboard cutout effect and so on issuppressed and a sense of three-dimensionality/sense of thickness isrestored to the subject.

Note that the “smoothing process” refers to a process for smoothing asignal waveform of an image signal. The “smoothing process” includes,for example, (1) a process for removing high-frequency components, (2) aprocess for removing shooting effect components, (3) a process forremoving ringing effect components, (4) a process for removing jaggycomponents, and so on from the image signal.

According to a second aspect of the invention, the end part regiondetection unit includes a depth obtainment unit that obtains, from theleft eye image and the right eye image, a left eye distance image and aright eye distance image by obtaining distance information in athree-dimensional space for the subject contained in the left eye imageand the right eye image.

Furthermore, the end part region detection unit detects the end partregion of the subject in the left eye image and/or the right eye imagebased on the distance information of the subject obtained by the depthobtainment unit.

According to this three-dimensional image processing apparatus, the endpart region of the subject is detected in the left eye image and/or theright eye image based on the distance information (depth value) of thesubject, and a smoothing process (for example, a process for removinghigh-frequency components) is executed on at least one end of thedetected end part region of the subject. As a result, according to thisthree-dimensional image processing apparatus, it is possible to suitablyprevent a drop in the quality of a sense of three-dimensionality/senseof depth (for example, a drop in quality caused by a cardboard cutouteffect, a phenomenon in which frosted glass appears to follow thesubject, and so on) arising due to the end part region of the subjectinappropriately undergoing disparity matching in the left eye image andthe right eye image.

Accordingly, a three-dimensional image obtained by the three-dimensionalimage processing apparatus is a high-quality three-dimensional image inwhich the occurrence of the cardboard cutout effect and so on issuppressed and a sense of three-dimensionality/sense of thickness isrestored to the subject.

Note that “distance information in a three-dimensional space” refers to,for example, a distance from a point (an image capturing point) in athree-dimensional space that corresponds to a first point of view (forexample, a left eye point of view when obtaining the left eye image) ora second point of view (for example, a right eye point of view whenobtaining the right eye image) from which it is assumed the left eyeimage or the right eye image is captured in three dimensions, to a pointin the three-dimensional space (an image capturing space in which it isassumed the left eye image or the right eye image is captured in threedimensions) that corresponds to a first pixel that is a pixel in theleft eye image and a second pixel that is a pixel in the right eye imageand that corresponds to the first pixel.

Meanwhile, “distance information in a three-dimensional space for thesubject” (distance information that is information regarding thedistance of the subject) refers to information that has correlation withthe subject distance.

The “subject distance” refers to the distance from an object that isfocused upon the surface of an image sensor in an image capturing unit(for example, a CCD image sensor, a CMOS image sensor, or the like) to acamera (that is, the three-dimensional image capturing apparatus), andincludes the concepts of object point distance and conjugate distance(distance between objects). Furthermore, the “subject distance” is aconcept including the approximate distance from the three-dimensionalimage capturing apparatus to the subject, and is a concept including,for example, (1) the distance from the center of gravity of the overalllens (a first point of view lens and/or a second point of view lens) inthe optical system of the three-dimensional image capturing apparatus tothe subject, (2) the distance from the surface of the image sensor inthe image capturing unit to the subject, (3) the distance from thecenter of gravity (or the center) of the three-dimensional imagecapturing apparatus to the subject, (4) the distance from a line segmentthat connects the first point of view and the second point of view tothe subject, and so on.

According to a third aspect of the invention, the end part regiondetection unit includes an edge extraction unit that extracts, from theleft eye image and the right eye image, an edge of the subject containedin the left eye image and the right eye image.

Furthermore, the end part region detection unit detects the end partregion of the subject in the left eye image and/or the right eye imagebased on edge information of the left eye image and the right eye imageextracted by the edge extraction unit.

According to this three-dimensional image processing apparatus, the endpart region of the subject is detected in the left eye image and/or theright eye image based on the edge of the subject extracted by the edgeextraction unit, and a smoothing process (for example, a process forremoving high-frequency components) is executed on at least one end ofthe detected end part region of the subject. As a result, according tothis three-dimensional image processing apparatus, it is possible tosuitably prevent a drop in the quality of a sense ofthree-dimensionality/sense of depth (for example, a drop in qualitycaused by a cardboard cutout effect, a phenomenon in which frosted glassappears to follow the subject, and so on) arising due to the end partregion of the subject inappropriately undergoing disparity matching inthe left eye image and the right eye image

Accordingly, a three-dimensional image obtained by the three-dimensionalimage processing apparatus is a high-quality three-dimensional image inwhich the occurrence of the cardboard cutout effect and so on issuppressed and a sense of three-dimensionality/sense of thickness isrestored to the subject.

According to a fourth aspect of the invention, the end part regiondetection unit detects the region including an edge on the left side ofthe subject as a left end part region and furthermore detects the regionincluding an edge on the right side of the subject as a right end partregion.

Furthermore, the edge correction unit:

(1) executes a smoothing process on the left end part region in the lefteye image; and

(2) executes a smoothing process on the right end part region in theright eye image.

According to this three-dimensional image processing apparatus, asmoothing process (for example, a process for removing high-frequencycomponents) is executed on regions that easily become occluded regions,and thus edge areas are blurred in regions that easily become occludedregions. As a result, according to this three-dimensional imageprocessing apparatus, it is possible to suitably prevent a drop in thequality of a sense of three-dimensionality/sense of depth (for example,a drop in quality caused by a cardboard cutout effect, a phenomenon inwhich frosted glass appears to follow the subject, and so on) arisingdue to the end part region of the subject inappropriately undergoingdisparity matching in the left eye image and the right eye image.

Accordingly, a three-dimensional image obtained by the three-dimensionalimage processing apparatus is a high-quality three-dimensional image inwhich the occurrence of the cardboard cutout effect and so on issuppressed and a sense of three-dimensionality/sense of thickness isrestored to the subject.

According to a fifth aspect of the invention, the end part regiondetection unit detects the region including an edge on the left side ofthe subject as a left end part region and furthermore detects the regionincluding an edge on the right side of the subject as a right end partregion.

Furthermore, the edge correction unit:

(1) executes a smoothing process on the left end part region in the lefteye image at a first strength and executes a smoothing process on theright end part region in the left eye image at a second strength that isa lower strength than the first strength; and

(2) executes a smoothing process on the right end part region in theright eye image at a third strength and executes a smoothing process onthe left end part region in the right eye image at a fourth strengththat is a lower strength than the third strength.

According to this three-dimensional image processing apparatus, astronger smoothing process (for example, a process for removinghigh-frequency components) is executed on regions that easily becomeoccluded regions, and thus edge areas are blurred more strongly inregions that easily become occluded regions than edge areas in regionsthat do not easily become occluded regions. As a result, according tothis three-dimensional image processing apparatus, it is possible tosuitably prevent a drop in the quality of a sense ofthree-dimensionality/sense of depth (for example, a drop in qualitycaused by a cardboard cutout effect, a phenomenon in which frosted glassappears to follow the subject, and so on) arising due to the end partregion of the subject inappropriately undergoing disparity matching inthe left eye image and the right eye image.

Accordingly, a three-dimensional image obtained by the three-dimensionalimage processing apparatus is a high-quality three-dimensional image inwhich the occurrence of the cardboard cutout effect and so on issuppressed and a sense of three-dimensionality/sense of thickness isrestored to the subject.

A sixth aspect of the invention is a three-dimensional image processingmethod that performs an image correction process on a left eye image anda right eye image contained in a three-dimensional image obtained by adual-lens technique or a multiple-viewpoint technique. The followingprocesses are executed in the three-dimensional image processing method.

(1) A process for detecting, from the left eye image and the right eyeimage, one or both of a region including an edge on a left side of asubject contained in the left eye image and the right eye image and aregion including an edge on a right side of the subject contained in theleft eye image and the right eye image, as an end part region.

(2) A process for smoothing a region in at least one end part region ofthe subject that has been detected.

Through this, it is possible to achieve an image processing method thatachieves the same effects as the first aspect of the invention.

According to the present invention, a sense of three-dimensionality andthickness can be restored to a subject and a high-qualitythree-dimensional image with a low sense of a cardboard cutout effectcan be obtained, regardless of the cause of the cardboard cutout effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the overall configuration of athree-dimensional image capturing apparatus 1000 according to a firstembodiment, including an image capturing environment (a captured scene).

FIG. 2 is a block diagram illustrating an image correction unit 104according to the first embodiment.

FIG. 3A, FIG. 3B, and FIG. 3C are a descriptive diagram illustrating animage capturing environment and a subject.

FIG. 4A, FIG. 4B, and FIG. 4C are a diagram illustrating a method forobtaining first depth information (R depth information) and second depthinformation (L depth information) through disparity matching

FIG. 5 is a diagram illustrating a signal waveform in a strengthgeneration unit 132 according to the first embodiment.

FIG. 6 is a flowchart illustrating processing performed by the strengthgeneration unit according to the first embodiment.

FIG. 7 is a diagram illustrating a result of processing performed by athree-dimensional image capturing apparatus according to the firstembodiment.

FIG. 8 is a diagram illustrating an example of a processing result inwhich an edge has been excessively blurred in a three-dimensional imageobtained by capturing a captured scene 200 three-dimensionally.

FIG. 9 is a diagram illustrating a signal waveform of a strengthgeneration unit 132 according to a variation on the first embodiment.

FIG. 10A and FIG. 10B are flowcharts illustrating processing performedby the strength generation unit according to the variation on the firstembodiment.

FIG. 11 is a diagram illustrating a result of processing performed by athree-dimensional image capturing apparatus according to a variation onthe first embodiment.

FIG. 12 is a diagram illustrating a signal waveform in a strengthgeneration unit 132 according to a variation on the first embodiment.

FIG. 13 is a block diagram illustrating the overall configuration of athree-dimensional image capturing apparatus 1000 according to a secondembodiment.

FIG. 14 is a block diagram illustrating an image correction unit 201according to the second embodiment.

DETAILED DESCRIPTION

Embodiments of the present invention will be described hereinafter withreference to the drawings. It will be apparent to those skilled in theart from this disclosure that the following descriptions of theembodiments are provided for illustration only and not for the purposeof limiting the invention as defined by the appended claims and theirequivalents

First Embodiment

The first embodiment will describe a dual-lens three-dimensional imagecapturing apparatus (digital camera, video camera, or the like) as anexample of a three-dimensional image processing apparatus.

Generally speaking, an R image and an L image obtained by capturing asubject three-dimensionally are obtained after first executing aperturecorrection (edge enhancement) on those images during camera signalprocessing. In normal two-dimensional images, a slight ringing effect(due to overshoot, undershoot, or the like) has little negativeinfluence on the image quality and can increase a sense of highresolution in the two-dimensional image, and is therefore actively usedas an image quality enhancement process in two-dimensional images.

However, if this processing, which is useful for two-dimensional images,is carried out on a three-dimensional image (R image and L image)obtained through three-dimensional imaging by a three-dimensional imagecapturing apparatus, the processed three-dimensional image (R image andL image) will have unnatural edge lines surrounding the target object asa subject, and can consequently appear to be a synthesized image. As aresult, a three-dimensional image (R image and L image) on which suchprocessing has been carried out becomes a three-dimensional image havingan extremely strong cardboard cutout effect.

Even in the case where there is no clear ringing effect in athree-dimensional image (R image and L image), the image willnevertheless have a strong cardboard cutout effect when thethree-dimensional image (R image and L image) is displayedthree-dimensionally in the case where the three-dimensional image (Rimage and L image) contains strong edges having what is known as binarycontrast, where the definition is high and the edges have a sharpluminance curve, the case where a process for enhancing the definitionhas been performed on the three-dimensional image (R image and L image)and as a result the three-dimensional image (R image and L image)contains edges with jaggies due to the occurrence of aliasing, or othersuch cases.

As is clear from these descriptions, the demands on the edges of atarget object differ between three-dimensional images andtwo-dimensional images.

The present embodiment provides a solution for the aforementionedproblems. In other words, the present embodiment aims to solve theaforementioned problems by performing, on a three-dimensional image (Rimage and L image), a process for smoothing edges in a border region ofa target object, a process for eliminating jaggies, a process foreliminating a ringing effect in the vicinity of the edges, eliminatingshooting effects (overshoot, undershoot, and so on), and the like andcorrecting the edges so that the edges are more suitable for athree-dimensional image.

1.1: Configuration of Three-Dimensional Image Capturing Apparatus

FIG. 1 is a schematic diagram illustrating the configuration of athree-dimensional image capturing apparatus 1000 according to the firstembodiment. Note that FIG. 1 schematically illustrates both thethree-dimensional image capturing apparatus 1000 and a scene 200captured by the three-dimensional image capturing apparatus 1000 (thatis, a captured scene 200).

As shown in FIG. 1, the three-dimensional image capturing apparatus 1000includes a first image capturing unit 101 R that obtains a first imagesignal (for example, a right eye image signal (an R image signal)) bycollecting subject light from a first point of view, a second imagecapturing unit 101 L that obtains a second image signal (for example, aleft eye image signal (an L image signal) by collecting subject lightfrom a second point of view, and an image input unit 102 that convertsthe first image signal (for example, the R image signal) and the secondimage signal (for example, the L image signal) into respective digitalsignals.

In addition, the three-dimensional image capturing apparatus 1000includes a depth obtainment unit 103 that calculates subject distanceinformation based on the respective first image signal (for example, theR image signal) and second image signal (for example, the L imagesignal) converted into digital signals and outputs the calculatedinformation as first depth information (for example, R depthinformation) and second depth information (for example, L depthinformation), and an image correction unit 104 that performs an imagecorrection process on the first image signal (for example, the R imagesignal) and the second image signal (for example, the L image signal)using the first depth information (for example, R depth information) andthe second depth information (for example, L depth information).

In addition, the three-dimensional image capturing apparatus 1000includes, as shown in FIG. 1, a control unit 105 that controls theaforementioned functional units. Note that the control unit 105 may beconnected directly to the functional units of the three-dimensionalimage capturing apparatus 1000, or may be connected to the functionalunits via a bus.

Note that for the sake of simplicity, the following descriptions will begiven assuming that a right eye image (video) is captured by the firstimage capturing unit 101R and a left eye image (video) is captured bythe second image capturing unit 101L.

The first image capturing unit 101R includes an optical system disposedat the first point of view that collects subject light and an imagesensor that obtains the first image signal (right eye image signal (Rimage signal)) by photoelectrically converting the collected subjectlight. The first image capturing unit 101 R then outputs the obtainedfirst image signal (R image signal) to the image input unit 102.

The second image capturing unit 101L includes an optical system disposedat the second point of view, corresponding to a different location thanthe first point of view, that collects subject light and an image sensorthat obtains the second image signal (left eye image signal (L imagesignal)) by photoelectrically converting the collected subject light.The second image capturing unit 101L then outputs the obtained secondimage signal (L image signal) to the image input unit 102.

The image input unit 102 is inputted with the first image signal (Rimage signal) obtained by the first image capturing unit 101R, performsA/D conversion on the inputted first image signal, and outputs theA/D-converted first image signal (R image signal) to the depthobtainment unit 103 and the image correction unit 104.

The image input unit 102 is also inputted with the second image signal(L image signal) obtained by the second image capturing unit 101 L,performs AID conversion on the inputted second image signal, and outputsthe A/D-converted second image signal (L image signal) to the depthobtainment unit 103 and the image correction unit 104.

The depth obtainment unit 103 is inputted with the first image signal (Rimage signal) and the second image signal (L image signal) outputtedfrom the image input unit 102. From a first image (R image) formed basedon the first image signal (R image signal) and a second image (L image)formed based the second image signal (L image signal), the depthobtainment unit 103 obtains the first depth information (R depthinformation) that is depth information of the first image (R image) andthe second depth information (L depth information) that is depthinformation of the second image (L image). The depth obtainment unit 103then outputs the obtained first depth information (R depth information)and second depth information (L depth information) to the imagecorrection unit 104.

Note that it is preferable for the depth information to be obtainedthrough, for example, disparity matching.

As shown in FIG. 2, the image correction unit 104 includes an L imagecorrection unit 104L and an R image correction unit 104R. The imagecorrection unit 104 is inputted with the first image signal (R imagesignal) and the second image signal (L image signal) outputted from theimage input unit 102, and the first depth information (R depthinformation) and the second depth information (L depth information)outputted from the depth obtainment unit 103. The image correction unit104 carries out a correction process on the first image signal (R imagesignal) based on the first depth information (R depth information) andoutputs the corrected first image signal (R image signal). Furthermore,the image correction unit 104 carries out a correction process on thesecond image signal (L image signal) based on the second depthinformation (L depth information) and outputs the corrected second imagesignal (L image signal).

As shown in FIG. 2, the L image correction unit 104L includes an edgecorrection unit 13 and the strength generation unit 132.

The edge correction unit 13 includes a smoothing unit 131 and asynthesizing unit 133, as shown in FIG. 2.

The smoothing unit 131 is inputted with a second image signal (L imagesignal) IS_L outputted from the image input unit 102, and performs asmoothing process on the inputted second image signal (L image signal)IS_L. The smoothing unit 131 then outputs a smoothed second image signal(L image signal) SS_L to the synthesizing unit 133.

The synthesizing unit 133 is inputted with the second image signal (Limage signal) IS_L outputted from the image input unit 102, the smoothedsecond image signal (L image signal) SS_L outputted from the smoothingunit 131, and a strength signal K1_L outputted from the strengthgeneration unit 132. The synthesizing unit 133 synthesizes the secondimage signal (L image signal) IS_L and the smoothed second image signal(L image signal) SS_L based on the strength signal K1_L, and outputs thesynthesized signal as a second image signal (L image signal) Lout.

The strength generation unit 132 is inputted with the second depthinformation (L depth information) outputted from the depth obtainmentunit 103, and generates the strength signal K1_L from the second depthinformation (L depth information). The strength generation unit 132 thenoutputs the generated strength signal K1_L to the synthesizing unit 133of the edge correction unit 13.

The L image correction unit 104L is thus configured in such a manner.

Note that the R image correction unit 104R has a similar configurationas the L image correction unit 104L, and differs from the L imagecorrection unit 104L only in that the inputted signals are the R imagesignal and the R depth information.

As shown in FIG. 1, the control unit 105 is connected to the first imagecapturing unit 101R, the second image capturing unit 101L, the imageinput unit 102, the depth obtainment unit 103, and the image correctionunit 104 so as to be capable of exchanging required signals in bothdirections therewith. The control unit 105 controls the aforementionedfunctional units of the three-dimensional image capturing apparatus 1000using predetermined control signals (driving signals, synchronizationsignals, and so on) so that signal processes on the R image signal andthe L image signal, data read/write processes, and so on are executed atpredetermined timings.

Note that an “end part region detection unit” is realized by the depthobtainment unit 103 and the strength generation unit 132 of the L imagecorrection unit 104L when processing the L image and by the depthobtainment unit 103 and the strength generation unit 132 of the R imagecorrection unit 104R when processing the R image.

1.2: Operations of Three-Dimensional Image Capturing Apparatus

Operations of the three-dimensional image capturing apparatus 1000configured as described thus far will be described hereinafter.

In FIG. 1, the captured scene 200 includes a far scene 201 and a nearscene 202. The near scene 202 corresponds to a primary subject.Operations performed by the three-dimensional image capturing apparatus1000 using the case where the three-dimensional image capturingapparatus 1000 captures the captured scene shown in FIG. 1 (the capturedscene shown in FIG. 3) in three dimensions as an example will now bedescribed.

FIG. 3 is a diagram schematically illustrating (an example of) arelationship between an image capturing environment and a subject in thecase where three-dimensional image capturing is carried out using thethree-dimensional image capturing apparatus 1000, illustrating thecaptured scene shown in FIG. 1 from above. FIG. 3A is a diagramillustrating the image capturing environment (captured scene) 200, andthe first image capturing unit 1018 and the second image capturing unit101L, from above. In the image capturing environment (captured scene)200, a primary subject 202 in the near scene and a subject 201 in thefar scene are in a positional relationship such as that illustrated inFIG. 3A. Although the subject 201 in the far scene is illustrated, forthe sake of simplicity, as being a wall or the like on which a pictureis drawn, it should be noted that the subject is not limited to such asubject, and it goes without saying that the subject may be anyexemplary subject.

FIG. 3B illustrates a luminance distribution of the picture drawn on thesubject 201 in the far scene, whereas FIG. 3C illustrates a frontalluminance distribution of the primary subject 202 in the near scene asseen from the three-dimensional image capturing apparatus 1000 (acamera).

Note that in FIGS. 3B and (c), the horizontal axis represents a positionin the horizontal direction, whereas the vertical axis representsluminance.

Note that for the sake of simplicity, an angle of convergence is set sothat a center line of the angle of view captured by the first imagecapturing unit 101R of the three-dimensional image capturing apparatus1000 (that is, a dot-dash line extending from 101R in FIG. 3A) and acenter line of the angle of view captured by the second image capturingunit 101 L (a dot-dash line extending from 101L in FIG. 3A) intersect ata distance (d2) at which the far scene 201 is located.

In addition, it is assumed that the near scene 202 (primary subject 202)is, for example, an object having a three-dimensional roundness (forexample, an approximately oval-shaped object having a predeterminedwidth when viewed from above (such as a person)).

Furthermore, although the angle of convergence has been set as describedabove for the sake of simplicity, the angle of convergence is notlimited thereto and may be set to another angle.

Subject light from the captured scene 200 is collected by the firstimage capturing unit 101 R disposed at the first point of view, and isconverted into the first image signal (R image signal) by the imagesensor in the first image capturing unit 101R. Likewise, subject lightfrom the captured scene 200 is collected by the second image capturingunit 101L disposed at the second point of view, and is converted intothe second image signal (L image signal) by the image sensor in thesecond image capturing unit 101L.

Note that the first image capturing unit 101R and the second imagecapturing unit 101L are disposed at a distance equivalent to aninteraxial distance (stereo base length) so that the three-dimensionalimage capturing apparatus 1000 can obtain a three-dimensional image (aleft eye image and a right eye image).

The first image signal (R image signal) outputted from the first imagecapturing unit 101R and the second image signal (L image signal)outputted from the second image capturing unit 101L are respectivelyinputted into the image input unit 102 and converted into digitalsignals by the image input unit 102. The first image signal (R imagesignal) and second image signal (L image signal) that have beenconverted into digital signals are then outputted to the depthobtainment unit 103 and the image correction unit 104.

From the first image (R image) formed based on the first image signal (Rimage signal) and the second image (L image) formed based the secondimage signal (L image signal), the depth obtainment unit 103 obtains thefirst depth information (R depth information) that is depth informationof the first image (R image) and the second depth information (L depthinformation) that is depth information of the second image (L image)through, for example, disparity matching

Here, a method for obtaining the first depth information (R depthinformation) and the second depth information (L depth information)through disparity matching will be described using FIG. 4.

FIG. 4 is a diagram schematically illustrating a three-dimensional imageproduced when capturing a captured scene, in which a triangular objectis disposed in the far scene and a circular object is disposed in thenear scene, in three dimensions using the three-dimensional imagecapturing apparatus 1000. FIG. 4A is a diagram schematicallyillustrating the L image (left eye image), FIG. 4B is a diagramschematically illustrating the R image (right eye image), and FIG. 4C isa diagram illustrating the R image and the L image being overlapped anddisplayed as a single image.

The method for obtaining the first depth information (R depthinformation) and the second depth information (L depth information)through disparity matching is realized, for example, by executing theprocesses described in the following (1) through (3).

(1) First, the depth obtainment unit 103 uses the L image (left eyeimage) and the R image (right eye image) to detect that, for example, asubject A corresponding to a point AL in the L image shown in FIG. 4A(that is, the apex of the triangle in FIG. 4) corresponds to a point ARin the R image shown in FIG. 4B.

(2) Then, a skew amount (disparity) Diff(A) between the two detectedpoints, or the point AL and the point AR, is calculated.

Note that the disparity has a positive/negative symbol depending on thedirection of the skew. This is, for example, positive in the case wherethe point in the R image is skewed to the left relative to the point inthe L image, and negative when the reverse is true.

For example, in the case of FIG. 4, if it is assumed that an absolutevalue of the disparity for the subject A is α (≧0), the AR point in theR image is skewed to the right of the AL point in the L image, and thusthe disparity for the subject A is calculated as “−α”. Likewise, if itassumed that an absolute value of the disparity for a subject B (thecenter point of the circle in FIG. 4) is β (≧, 0), a BR point in the Rimage is skewed to the left of a BL point in the L image, and thus thedisparity for the subject B is calculated as “+β”.

(3) The depth obtainment unit 103 carries out the processes of (1) and(2) for all points (all pixels) in the image, and obtains a disparityimage that takes the calculated skew amounts (disparities) as pixelvalues. Then, a disparity image obtained using the disparitiescalculated for the respective pixels in the L image as pixel values istaken as the L depth information (an L depth information image (a lefteye image distance image)), and a disparity image obtained using thedisparities calculated for the respective pixels in the R image as pixelvalues is taken as the R depth information (an R depth information image(a right eye image distance image)).

For example, with the L depth information (L depth information image(left eye image distance image)), the value of the pixel correspondingto the AL point in the L image shown in FIG. 4A is −α, which is thedisparity of the subject A, whereas with the R depth information (Rdepth information image (right eye image distance image)), the value ofthe pixel corresponding to the AR point in the R image shown in FIG. 4Bis −α, which is the disparity of the subject A.

Note that “distance image” refers to an image in which for each pixel, avalue having correlation with the distance between the actual locationof the subject corresponding to each pixel (that is, a location within athree-dimensional space) and the location of the three-dimensional imagecapturing apparatus 1000 is mapped.

Note that the method for obtaining the first depth information (R depthinformation) and the second depth information (L depth information)through disparity matching is merely an example, and the method is notlimited thereto. For example, the stated symbols for the disparities maybe reversed. In addition, the depth obtainment unit 103 may obtain theleft eye image distance image and the right eye image distance image,and may obtain the L depth information and the R depth information,using another method.

The L depth information and the R depth information obtained asdescribed above are respectively outputted to the image correction unit104.

1.2.1: Operations of Image Correction Unit 104

Next, operations performed by the image correction unit 104 will bedescribed.

The L image correction unit 104L executes processing on the L imageusing the L depth information and the R image correction unit 1048executes processing on the R image using the R depth information.Because the details of those processes are the same, the followingdescriptions will focus on the L image correction unit 104L.

First, operations performed by the strength generation unit 132 of the Limage correction unit 104L will be described.

FIG. 5 is a diagram illustrating operations performed by the strengthgeneration unit 132. In the uppermost section of FIG. 5, the horizontalaxis represents a location in the horizontal direction and the verticalaxis represents a value of the L depth information. In other words, DLin the uppermost section of FIG. 5 indicates depth information (distanceinformation) of the L image for a pixel location in the L image, andexpresses the depth information (distance information) of the L imageprovided from the depth obtainment unit 103. Here, DL takes on a lowervalue for further distances (that is, greater subject distances), andtakes on a higher value for closer distances (that is, lower subjectdistances).

Accordingly, as shown in the uppermost section of FIG. 5, the depthinformation (distance information) of the primary subject 202 has ahigher value than the depth information (distance information) of thefar scene subject 201. Meanwhile, as shown in the uppermost section ofFIG. 5, the value of the L depth information of the primary subject 202takes on a value in a range from D2 _(—) f to D2 _(—) n. In other words,the value of the L depth information at the closest point of the primarysubject 202 is D2 _(—) n, and the value of the L depth information atthe farthest point of the primary subject 202 is D2 _(—) f. Meanwhile,because the far scene subject 201 is a flat wall, the L depthinformation of the far scene subject 201 is a constant D1.

Operations performed by the strength generation unit 132 will bedescribed using FIG. 5 and the flowchart shown in FIG. 6.

(S101, S102):

The strength generation unit 132 obtains an edge-corrected L depth valueDL′ (this corresponds to a curve Crv1 in FIG. 5) by performing an edgecorrection process (for example, an LPF process) on the inputted L depthinformation (L depth value) (that is, by performing a smoothingprocess).

(S103):

The strength generation unit 132 obtains an L depth differential signalΔDL by finding differentials (differences) for the horizontal locations(that is, the values in the X axis direction in FIG. 5) of theedge-corrected L depth value DL′. Note that the process of S103 needsnot be a process for finding differentials (differences); as long as theprocess can detect an amount of change in the edge-corrected L depthvalue DL′, the process may be executed instead of a process for findingdifferentials (differences).

(S104):

The strength generation unit 132 obtains an L depth differentialabsolute value signal ΔDL1 by finding the absolute value of the L depthdifferential signal ΔDL.

(S105):

The strength generation unit 132 outputs, to the edge correction unit 13(the synthesizing unit 133), a signal obtained by normalizing theobtained L depth differential absolute value signal ΔDL1 (for example, asignal normalized to a range of [0:1]) as the L image strength signalK1_L.

Through this, the three-dimensional image capturing apparatus 1000 canaccurately detect the vicinity of edges having different depth values(that is, edges formed by subjects having different depth values) usingthe L image strength signal K1_L generated by the strength generationunit 132.

Note that the bandwidth of the L depth differential absolute valuesignal ΔDL1 that detects a region including edges (this corresponds tobands indicated by P1 and P2 in FIG. 5, and in the case of an R depthdifferential absolute value signal ΔDR1, corresponds to bands indicatedby Q1 and Q2 in FIG. 5) can be adjusted using the properties of alow-pass filter applied to the L depth signal (L depth value) DL.

Next, operations performed by the edge correction unit 13 of the L imagecorrection unit 104L will be described.

The smoothing unit 131 of the edge correction unit 13 executes asmoothing process on the L image signal IS_L outputted from the imageinput unit 102. In the present embodiment, the smoothing unit 131 is,for example, a low-pass filter, and executes an LPF process as thesmoothing process. Through this, the L image signal SS_L processed bythe smoothing unit 131 becomes a signal in which edge parts have beensmoothed (a smoothed signal (for example, a signal from whichhigh-frequency components have been removed, a signal from whichshooting effects have been removed, a signal from which ringing effectshave been removed, a signal from which jaggies have been removed, and soon)).

Note that the smoothing process performed by the smoothing unit 131 isnot limited to an LPF process, and may be, for example, a processperformed using a median filter. In other words, any processing thattends to smooth the edges may be executed by the smoothing unit 131 asthe smoothing process, and as long as the smoothing process performed bythe smoothing unit 131 is a process that can execute such processing(that is, processing that tends to blur the edges), any process may beused. Note that it is effective to employ processing combining a medianfilter, useful in removing fine jaggies having large changes inamplitude, with a standard low-pass filter, as the smoothing processperformed by the smoothing unit 131.

The L image signal SS_L on which the smoothing unit 131 has executed thesmoothing process is outputted to the synthesizing unit 133.

The synthesizing unit 133 synthesizes the L image signal IS_L and the Limage signal SS_L on which the smoothing process has been executedusing, for example, the L image strength signal K1_L, which is a signalthat has been normalized in the range of [0:1], as an internal divisionratio, and consequently obtains an output L image signal Lout. In otherwords, the synthesizing unit 133 obtains the output L image signal Loutby executing a process corresponding to:

Lout=(1−K1_(—) L)×IS _(—) L+K1_(—) L×SS _(—) L

Through this, the L image signal Lout outputted from the synthesizingunit 133 becomes (1) a smoothed image signal at edge parts formedbetween subjects having different depth values (that is, an image signalon which a smoothing process has been executed) and (2) an image signaloutputted from the image input unit in other areas (that is, an imagesignal on which a smoothing process has not been executed).

Note that the same processing as described above is executed on the Rimage signal as well, by the R image correction unit 104R.

As a result, the three-dimensional image processed by thethree-dimensional image capturing apparatus 1000 becomes athree-dimensional image in which only areas near edges are selectivelyblurred.

Note that the amount of the blurring obtained through the smoothingprocess performed by the smoothing unit 131 (for example, a low-passfiltering process) is set in accordance with the value of the strengthsignal K1_L. Accordingly, because the amount of the smoothing increasestoward regions in the vicinity of edges, a three-dimensional image fromwhich jaggies, shooting effects (overshoot, undershoot, and the like),and so on have been effectively removed from edge regions can beobtained through the processing performed by the three-dimensional imagecapturing apparatus 1000.

A processing result obtained in the case where the three-dimensionalimage capturing apparatus 1000 has performed the aforementionedprocessing is illustrated in FIG. 7.

As can be seen from FIG. 7, in areas near the left and right edges ofthe primary subject 202 in the L image (the regions indicated by P1 andP2 in FIG. 7), a smoothing process is executed at a strength based onthe strength signal K1_L (the value of ΔDL1) (that is, a high-strengthsmoothing process is executed in the vicinity of locations L1 and L2).

Meanwhile, in areas near the left and right edges of the primary subject202 in the R image (the regions indicated by Q1 and Q2 in FIG. 7), asmoothing process is executed at a strength based on a strength signalK1_R (the value of ΔDR1) (that is, a high-strength smoothing process isexecuted in the vicinity of locations R1 and R2).

The three-dimensional image (L image and R image) whose edges have beenappropriately corrected by the three-dimensional image capturingapparatus 1000 according to the present embodiment in this manner is athree-dimensional image in which unnaturalness at the edges of thesubject has been reduced, and when displayed in three dimensions,elements that are recognized as strange synthesized images have beeneffectively reduced. As a result, the three-dimensional image obtainedby the three-dimensional image capturing apparatus 1000 is a natural,high-quality three-dimensional image, with little cardboard cutouteffect, when displayed in three dimensions.

Although the above describes an example in which the edge correctionunit 13 is configured of the smoothing unit 131 and the synthesizingunit 133, it should be noted that the embodiment is not limited thereto,and the edge correction unit 13 may be realized, for example, as alow-pass filter with a variable cutoff frequency. In this case, thelow-pass filter with a variable cutoff frequency is controlled by thestrength signals K2 (K2_L or K2_R). In other words, in this case, theoutput image is obtained by executing a filtering process on the inputimages (L image or R image) having lowered the cutoff frequency only inthe vicinity of edge regions based on the strength signals K2 (K2_L orK2_R). Through this, only regions in the vicinity of edges in thethree-dimensional image (that is, edge regions formed by subjects havingdifferent depth values) can be selectively smoothed, and thus the sameeffects as those described above can be achieved.

Variation

Next, a variation on the present embodiment will be described.

The foregoing describes the three-dimensional image capturing apparatus1000 according to the first embodiment as being capable of correctingedge areas in a three-dimensional image to a natural state of blurrinessby removing ringing effects, jaggies, and the like in the edge areas byappropriately correcting the edges, thus being capable of obtaining athree-dimensional image in which a cardboard cutout effect has beenreduced.

However, when performing an image correction process on athree-dimensional image, it is not easy to set the amount of blurringappropriately; if the image correction process is carried out with a lowamount of blurring, the ringing effect and the like will remain, whereasif the image correction process is carried out with an excessive amountof blurring, the edge areas will be excessively blurred.

FIG. 8 is a diagram illustrating (an example of) a processing result inwhich an edge has been excessively smoothed in a three-dimensional imageobtained by capturing the scene 200 three-dimensionally.

As shown in FIG. 8, the shaded edge regions indicate ranges in which theedges have been excessively smoothed. An “excessively blurred state”refers to a signal expressing a luminance value of the end of theprimary subject 202 being smoothed with some of the luminance of the endof the primary subject 202 bleeding into the far scene 201 (for example,a loss of high-frequency components).

When such an image is viewed in three dimensions, the far scene inranges around the edges of the subject (ranges indicated by AL402,AL403, AR402, and AR403 in FIG. 8) are smoothed (for example, have aloss of high frequency components), and as a result, it becomesdifficult to use the pattern of the far scene for disparity matchingAccordingly, when the three-dimensional image shown in FIG. 8 isdisplayed in three dimensions, there will be a visual sense that theblurred far scenes around the edges of the L image and the R image arematched. As a result, even regions of the blurred far scene outside ofthe edge areas of the subject will be recognized as having the samedisparity as the subject (that is, will be recognized as having the samedisparity as the disparity of the primary subject 202), and the regionindicated by A′ in FIG. 8 will be recognized as part of the near scene(that is, as part of the primary subject 202).

If such a three-dimensional image is displayed in three dimensions,there will be a visual sense of frosted glass following the edge areasof the subject, in which a blurred far scene appears.

If part of the far scene follows the periphery of the near scene (theprimary subject 202) in this manner, when the image is displayed inthree dimensions, the near scene (the primary subject 202) will appearflat, exacerbating the cardboard cutout effect, in the same manner aswith a three-dimensional image in which a ringing effect has occurred.

In other words, in the case where a ringing effect is removed from edgesin order to reduce the cardboard cutout effect when performing an imagecorrection process on a three-dimensional image, the ringing effect willremain if the blurring is insufficient, whereas if the blurring isexcessive, a new cardboard cutout effect will be produced by the excessblurring. It is thus difficult to ensure a consistently stable processwhen performing image correction processing on three-dimensional images.

The present variation solves the aforementioned problems.

The configuration of the three-dimensional image capturing apparatusaccording to the present variation is the same as the configuration ofthe three-dimensional image capturing apparatus 1000 according to thefirst embodiment.

With the three-dimensional image capturing apparatus according to thepresent variation, the strength signal generated by the strengthgeneration unit 132 of the image correction unit 104 differs from thatin the three-dimensional image capturing apparatus 1000 according to thefirst embodiment.

The strength generation unit 132 of the L image correction unit 104L inthe three-dimensional image capturing apparatus according to the presentvariation outputs an L image second strength signal K2_L instead of theL image strength signal K1_L in the first embodiment.

The strength generation unit 132 of the R image correction unit 104R inthe three-dimensional image capturing apparatus according to the presentvariation outputs an R image second strength signal K2_R instead of theR image strength signal K1_R in the first embodiment.

Note that in the present variation, elements that are identical to thosein the three-dimensional image capturing apparatus 1000 according to thefirst embodiment are assigned the same reference numerals, and detaileddescriptions thereof will be omitted.

FIG. 9 is a waveform diagram illustrating processing performed by thestrength generation unit 132 according to the present variation, andillustrates the L depth signal (L depth value) DL, the edge-corrected Ldepth signal (edge-corrected L depth value) DL′, the L depthdifferential signal ΔDL, a second L depth differential absolute valuesignal ΔDL2, and a second R depth differential absolute value signalΔDR2.

Note that in FIG. 9, the L depth signal (L depth value) DL and the Ldepth differential signal ΔDL are L image signals, and are the same asthose shown in FIG. 5.

The second L depth differential absolute value signal ΔDL2 is a signalfor generating the L image second strength signal K2_L, and the L imagesecond strength signal K2_L is a signal in which the second L depthdifferential absolute value signal ΔDL2 has been normalized (forexample, a signal normalized in the range of [0:1]).

The second R depth differential absolute value signal ΔDR2 is a signalfor generating the R image second strength signal K2_R, and the R imagesecond strength signal K2_R is a signal in which the second R depthdifferential absolute value signal ΔDR2 has been normalized (forexample, a signal normalized in the range of [0:1]).

Meanwhile, the second L depth differential absolute value signal ΔDL2 isa signal obtained by removing areas in which the signal value isnegative and leaving only areas in which the signal value is positive inthe L depth signal ΔDL shown in FIG. 5 (the signal may include areas inwhich the signal value is 0).

The L image strength generation unit 132 generates the L image secondstrength signal K2_L by normalizing the second L depth differentialabsolute value signal ΔDL2 (for example, normalizing in the range of[0:1]), and outputs the generated L image second strength signal K2_L tothe L image edge correction unit 13.

Meanwhile, the second R depth differential absolute value signal ΔDR2 isa signal generated by the R image strength generation unit 132, and is asignal generated through the following processing.

(1) Taking an R depth differential signal ΔDR, which is the R imagesignal corresponding to the L depth differential signal ΔDL in FIG. 5,the signal value of that signal is positive-negative inverted.

(2) Areas in which the signal value is negative are removed and onlyareas in which the signal value is positive are left in the signalobtained through the process (1). Through this, the R image strengthgeneration unit 132 generates the R depth differential signal ΔDR.

The R image strength generation unit 132 generates the R image secondstrength signal K2_R by normalizing the second R depth differentialabsolute value signal ΔDR2 (for example, normalizing in the range of[0:1]), and outputs the generated R image second strength signal K2_R tothe R image edge correction unit 13.

As shown in FIG. 9, in the present variation, the L image secondstrength signal K2_L (second L depth differential absolute value signalΔDL2) is produced only in the vicinity of the edge area on the left endof the subject (the primary subject 202), whereas the R image secondstrength signal K2_R (second R depth differential absolute value signalΔDR2) is produced only in the vicinity of the edge area on the right endof the subject (the primary subject 202).

FIG. 10 is a flowchart illustrating processing performed by the strengthgeneration unit 132 according to the present variation. FIG. 10A is aflowchart illustrating processing performed by the L image strengthgeneration unit 132, whereas FIG. 10B is a flowchart illustratingprocessing performed by the R image strength generation unit 132.

The processes performed by the strength generation unit 132 according tothe present variation will be described using FIG. 10.

First, processes performed by the L image strength generation unit 132will be described using FIG. 10A.

(S201, S202):

The strength generation unit 132 obtains the edge-corrected L depthvalue DL′ (this corresponds to a curve Crv1 in FIG. 9) by performing anedge correction process (for example, an LPF process) on the inputted Ldepth information (L depth value) (that is, by performing a smoothingprocess).

(S203):

The strength generation unit 132 obtains the L depth differential signalΔDL by finding differentials (differences) for the horizontal locations(that is, the values in the X axis direction in FIG. 9) of theedge-corrected L depth value DL′. Note that the process of S203 needsnot be definitely a process for finding differentials (differences); aslong as the process can detect an amount of change in the edge-correctedL depth value DL′, the process may be executed instead of a process forfinding differentials (differences).

(S204):

The strength generation unit 132 obtains the second L depth differentialabsolute value signal ΔDL2 by removing the signal at areas where thesignal value is negative from the L depth differential signal ΔDL (thatis, setting the signal value of the areas where the signal value isnegative to 0).

(S205):

The strength generation unit 132 outputs, to the edge correction unit 13(the synthesizing unit 133), a signal obtained by normalizing theobtained second L depth differential absolute value signal ΔDL2 (forexample, a signal normalized to a range of [0:1]) as the L imagestrength signal K2_L.

Next, processes performed by the R image strength generation unit 132will be described using FIG. 10B.

(S301, S302):

The strength generation unit 132 obtains an edge-corrected R depth valueDR′ by performing an edge correction process (for example, an LPFprocess) on the inputted R depth information (R depth value) (that is,by performing a smoothing process).

(S303):

The strength generation unit 132 obtains the R depth differential signalΔDR by finding differentials (differences) for the horizontal locations(that is, the values in the X axis direction and FIG. 9) of theedge-corrected R depth value DR′. Note that the process of S303 needsnot be definitely a process for finding differentials (differences); aslong as the process can detect an amount of change in the edge-correctedR depth value DR′, the process may be executed instead of a process forfinding differentials (differences).

(S304):

The strength generation unit 132 (1) obtains a signal in which thesignal values of the R depth differential signal ΔDR have beenpositive-negative inverted. Then, (2) the strength generation unit 132obtains the second R depth differential absolute value signal ΔDR2 byremoving parts of the signal in which the signal value is negative fromthe positive-negative inverted signal (that is, setting the signal valueof the areas where the signal value is negative to 0).

(S305):

The strength generation unit 132 outputs, to the edge correction unit 13(the synthesizing unit 133), a signal obtained by normalizing theobtained second R depth differential absolute value signal ΔDR2 (forexample, a signal normalized to a range of [0:1]) as the R imagestrength signal K2_R.

Through this, the L image strength signal K2_L is generated by the Limage strength generation unit 132 according to the present variation,and the R image strength signal K2_R is generated by the R imagestrength generation unit 132 according to the present variation.

Then, the L image edge correction unit 13 blurs only edge areas on theleft of the subject (the primary subject 202) using the strength signalK2_L generated by the strength generation unit 132. In other words, thesynthesizing unit 133 generates the output L image signal Lout byexecuting a process corresponding to the following formula, using thestrength signal K2_L as an internal division ratio.

Lout=(1−K2_(—) L)×IS _(—) L+K2_(—) L×SS _(—) L

Meanwhile, the R image edge correction unit 13 blurs only edge areas onthe right of the subject (the primary subject 202) using the strengthsignal K2_R generated by the strength generation unit 132. In otherwords, the synthesizing unit 133 generates an output R image signal Routby executing a process corresponding to the following formula, using thestrength signal K2_R as an internal division ratio.

Rout=(1−K2_(—) R)×IS _(—) R+K2_(—) R×SS _(—) R

In other words, with the three-dimensional image capturing apparatusaccording to the present variation, (1) a process for blurring only theedge areas on the left of the subject (the primary subject 202) isexecuted in the L image and (2) a process for blurring only the edgeareas on the right of the subject (the primary subject 202) is executedin the R image, based on the function of the second strength signals K2(K2_L and K2_R).

FIG. 11 is a diagram showing waveforms obtained after the processingperformed by the three-dimensional image capturing apparatus accordingto the present variation.

As can be seen in FIG. 11, (1) in the L image, only the edge areas onthe left of the subject (the primary subject 202) are blurred, whereas(2) in the R image, only the edge areas on the right of the subject (theprimary subject 202) are blurred.

Accordingly, as described above, when a three-dimensional image obtainedby the three-dimensional image capturing apparatus according to thepresent variation is displayed in three dimensions, regions in thevicinity of the edges of the subject (the primary subject 202) are notinappropriately matched. As a result, the three-dimensional imageobtained by the three-dimensional image capturing apparatus according tothe present variation is a three-dimensional image in which theoccurrence of a cardboard cutout effect or the like is suppressed andthat is capable of appropriately reproducing a sense ofthree-dimensionality and a sense of depth.

Although the above descriptions discuss, for the sake of simplicity, acase in which only the edge areas on the left of the subject (theprimary subject 202) are blurred in the L image and only the edge areason the right of the subject (the primary subject 202) are blurred in theR image, it should be noted that the variation is not limited thereto.

For example, the three-dimensional image capturing apparatus accordingto the present variation may perform processing such as that describedbelow.

(1) In the L image, the edge areas on the left of the subject (theprimary subject 202) are highly blurred (blurred at a first strength)and the edge areas on the right of the subject (the primary subject 202)are more weakly blurred (blurred at a second strength that is weakerthan the first strength).

(2) In the R image, the edge areas on the right of the subject (theprimary subject 202) are highly blurred (blurred at a third strength)and the edge areas on the left of the subject (the primary subject 202)are more weakly blurred (blurred at a fourth strength that is weakerthan the third strength).

FIG. 12 is a diagram illustrating waveforms of the L depth signal (Ldepth value) DL, the edge-corrected L depth signal (the edge-corrected Ldepth value) DL′, the L depth differential signal ΔDL, the second Ldepth differential absolute value signal ΔDL2, and the second R depthdifferential absolute value signal ΔDR2, in the case with theaforementioned processing is performed.

As shown in FIG. 12,

(1) In the L image, the peak value of the strength signal K1_L in theedge area on the right of the subject (the primary subject 202) (thatis, the region P2) is k (where 0≦k≦1) times the peak value of thestrength signal K1_L in the edge area on the left of the subject (theprimary subject 202) (that is, the region P1).

(2) In the R image, the peak value of the strength signal K1_R in theedge area on the left of the subject (the primary subject 202) (that is,the region Q1) is m (where 0≦m≦1) times the peak value of the strengthsignal K1_R in the edge area on the right of the subject (the primarysubject 202) (that is, the region Q2).

Accordingly, with the three-dimensional image capturing apparatusaccording to the present variation, when the edge correction unit 13executes processing using the aforementioned strength signals K1,

(1) In the L image, the edge areas on the left of the subject (theprimary subject 202) can be highly blurred (blurred at a first strength)and the edge areas on the right of the subject (the primary subject 202)can be more weakly blurred (blurred at a second strength that is weakerthan the first strength), and

(2) In the R image, the edge areas on the right of the subject (theprimary subject 202) can be highly blurred (blurred at a third strength)and the edge areas on the left of the subject (the primary subject 202)can be more weakly bluffed (blurred at a fourth strength that is weakerthan the third strength).

By executing processing in the manner described above, in athree-dimensional image obtained by the three-dimensional imagecapturing apparatus according to the present variation, the region AR402in the R image is not significantly blurred; as a result, the blurredregion AL402 in the L image and the region AR402 in the R image are notmatched, and thus are not sensed as being in the near scene.

Meanwhile, the far scene of the blurred region AL402 in the L image isoriginally a region that is highly likely to be an occluded region (thatis, the far scene in the corresponding position is not present in the Rimage), and is thus a region that does not have a component for stereomatching and whose sense of distance is indefinite

Accordingly, in the case where a three-dimensional image that has beenprocessed as described above by the three-dimensional image capturingapparatus according to the present variation is displayed in threedimensions, a side effect caused by blurring the left end area of theobject serving a subject in the L image is difficult to perceive.

Likewise, in the three-dimensional image obtained by thethree-dimensional image capturing apparatus according to the presentvariation, the region AL403 in the L image is not significantly blurred;as a result, the blurred region AR403 in the R image and the regionAL403 in the L image are not matched, and thus are not sensed as beingin the near scene.

Furthermore, the far scene of the blurred region AR403 in the R image isalso originally a region that is highly likely to be an occluded region,and is thus a region that does not undergo stereo matching and whosesense of distance is indefinite.

Accordingly, in the case where a three-dimensional image that has beenprocessed as described above by the three-dimensional image capturingapparatus according to the present variation is displayed in threedimensions, a side effect caused by blurring the right end area of theobject serving a subject in the R image is difficult to perceive.

As described thus far, with the three-dimensional image capturingapparatus according to the present variation, as long as the state ofthe blurring (the extent of the blurring) on the left and right sides ofthe subject (for example, the primary subject 202) are set to bedifferent, it is possible to avoid unsuitable disparity matching (thatis, that unsuitable image regions will be recognized as the same subjectdue to disparity matching), and as a result, it is possible toappropriately prevent the occurrence of the cardboard cutout effect andthe like.

In other words, with the three-dimensional image capturing apparatusaccording to the present variation, it is simply necessary to set thestate of the blurring (the extent of the blurring) on the left and rightsides of the subject (for example, the primary subject 202) to bedifferent, and thus there is less of a demand for precision in afiltering process carried out during the smoothing process. In otherwords, with the three-dimensional image capturing apparatus according tothe present variation, the margin for the extent of the filteringprocess can be increased in the smoothing process. As a result, with thethree-dimensional image capturing apparatus according to the presentvariation, a simple and stable image correction process can be executedon the three-dimensional image.

Although the present embodiment illustrates an example in which theluminance of the primary subject 202 is higher than the luminance of thefar scene 201, it should be noted that the embodiment is not limitedthereto. That is, the details and effects of the processing performed bythe three-dimensional image capturing apparatus 1000 according to thepresent embodiment and the three-dimensional image capturing apparatusaccording to the variation on the present embodiment have no relation tothe luminance of the subject.

Second Embodiment

Next, a second embodiment will be described.

A three-dimensional image capturing apparatus (three-dimensional imageprocessing apparatus) 2000 according to the second embodiment is athree-dimensional image capturing apparatus (three-dimensional imageprocessing apparatus) having the same effects and purpose as thatdescribed in the first embodiment, and FIG. 13 illustrates the overallconfiguration of the three-dimensional image capturing apparatus 2000according to the present embodiment.

The three-dimensional image capturing apparatus 2000 according to thepresent embodiment has the configuration of the three-dimensional imagecapturing apparatus 1000 according to the first embodiment, except thatthe depth obtainment unit 103 has been replaced with an edge extractionunit 20 and the image correction unit 104 has been replaced with asecond image correction unit 21. These points represent the differencesbetween the three-dimensional image capturing apparatus 2000 accordingto the present embodiment and the three-dimensional image capturingapparatus 1000 according to the first embodiment, and aside from thesedifferences, the three-dimensional image capturing apparatus 2000according to the present embodiment is the same as the three-dimensionalimage capturing apparatus 1000 according to the first embodiment.

Note that in the present embodiment, elements that are identical tothose of the first embodiment are assigned the same reference numerals,and detailed descriptions thereof will be omitted.

The edge extraction unit 20 is inputted with the R image and the L imageoutputted from the image input unit 102. The edge extraction unit 20extracts edge components of the subject from the R image and the L imageoutputted from the image input unit 102. The edge extraction unit 20outputs information regarding the edge components extracted through aprocess that will be described later (that is, an L image common edgeimage) to a second strength generation unit 22 in an L image correctionunit 21L of the second image correction unit 21, and outputs informationregarding the edge components extracted through a process that will bedescribed later (that is, an R image common edge image) to a secondstrength generation unit 22 in an R image correction unit 21R of thesecond image correction unit 21.

Note that the “information regarding the extracted edge components”refers to a concept that includes, for example, an edgecomponent-extracted image obtained by extracting edge components from animage (for example, an edge-detected image), an image signal that formssuch an edge component-extracted image (for example, an edge-detectedimage), and so on.

The edge extraction unit 20 performs an edge extraction process bydetecting primarily an amount of change in pixel values in theside-to-side direction of an image (the horizontal direction of animage), and performs the edge extraction process through various knownmethods using second derivatives, Laplacians, and so on for the pixelvalues in the side-to-side direction (horizontal direction) of theimage. However, the edge extraction unit 20 may perform the edgeextraction process by detecting an amount of change in pixel values in atop-to-bottom direction of the image (the vertical direction of theimage), a diagonal direction, or the like, in addition to theside-to-side direction of the image (the horizontal direction of theimage). Furthermore, the direction for the detection of the amount ofchange in the pixel values for executing the edge extraction process isnot limited to a single direction, and may be a plurality of directions.

Meanwhile, more highly-precise edge extraction methods have beenproposed, and the edge extraction unit 20 may carry out the edgeextraction process using such methods. In sum, any method may be usedfor the edge extraction process performed by the edge extraction unit 20as long as it is a method that enables edges to be extracted.

Meanwhile, the edge extraction unit 20 includes a function forextracting only the edges of a subject for which the occurrence of thecardboard cutout effect is to be reduced (this subject is often theprimary subject). The edge extraction unit 20 shifts an edge image R (anedge image R obtained as a result of extracting the edge component fromthe R image) and an edge image L (an edge image L obtained as a resultof extracting the edge component from the L image) in the side-to-sidedirection (the horizontal direction of the image) so that the edges inthe R image and the edges in the L image of the primary subject, whichis specified using a method not described here, overlap with each other.At this time, the edge extraction unit 20 determines the extent to whichthe edges of the primary subject in the R image and in the L imageoverlap, using, for example, a known correlation determination method.

The edge extraction unit 20 extracts common areas of the edge image Rand the edge image L in a state in which the edges of the primarysubject in the edge image R and the edges of the primary subject in theedge image L overlap the closest. For example, in the case where theedges are expressed as a grayscale from 0 to 255 (where stronger edgeshave greater values) (that is, the case where the pixel values of theedge image R and the edge image L are from 0 to 255 (that is, are 8-bitdata)), it is possible to extract common edges by finding the minimumvalue between the pixel value in the edge image R and the pixel value inthe edge image L at the same coordinates in a state in which the edgesof the primary subject in the edge image R and the edges of the primarysubject in the edge image L overlap the closest. In other words,assuming that the pixel value of the edge image R at a coordinatelocation (x,y) is R1(x,y) and the pixel value of the edge image L at thecoordinate location (x,y) is L1(x,y) in an image in which the edge imageR and the edge image L have been superimposed so that the edges of theprimary subject in the edge image R and the edges of the primary subjectin the edge image L overlap the closest, a minimum value Min(x,y)between the pixel value in the edge image R and the pixel value in theedge image L is found through the following:

when R1(x,y)>L1(x,y),   (1)

Min(x,y)=L1(x,y)

when R1(x,y)≦L1(x,y),   (2)

Min(x,y)=R1(x,y)

Then, by taking the minimum value Min(x,y) obtained through the aboveprocessing as the pixel value at the coordinate location (x,y), theprocess for extracting the common edge areas can be carried out.

As a result of this processing, the positions of the edge image R andthe edge image L are shifted in an image in which the edge image R andthe edge image L have been superimposed so that the edges of the primarysubject in the edge image R and the edges of the primary subject in theedge image L overlap the closest, and thus edges in which the depthvalues differ from the primary subject are eliminated. (That is, thepositions of the edge image R and the edge image L are shifted in animage in which the edge image R and the edge image L have beensuperimposed so that the edges of the primary subject in the edge imageR and the edges of the primary subject in the edge image L overlap theclosest, and thus the stated minimum value Min(x,y) takes on a valuethat is close to 0; as a result, edges in which the depth values differfrom the primary subject are eliminated.)

If approximate depth information (an approximate depth value) is known,edges in which the depth values are different can be further eliminatedby the edge extraction unit 20 performing processing based on that depthinformation (those depth values).

It is desirable to furthermore eliminate edges estimated to be locatedwithin edge objects of the primary subject in the edges of the primarysubject that remain following the aforementioned processing. Forexample, it is also possible for the edge extraction unit 20 to performa process for leaving left end and right end edges of the primarysubject and remove edges located in regions therebetween, or forweakening those edge components (that is, reducing the pixel values ofareas that form those edges).

Furthermore, it is also possible for the edge extraction unit 20 toperform a process for finding the colors of the R image and the L imageand, for example, detecting changes in the color based on hue,saturation, brightness, or the like, leaving the edges in image regionsin which the color changes, and removing the edges in image regions inwhich the color does not change, or for weakening those edge components(that is, reducing the pixel values of areas that form those edges).

Furthermore, it is also possible for the edge extraction unit 20 toperform a person detection process, known as a more highly-precisemethod, and perform processing for removing edges aside from the ends ofthe person by referring to the result of the person detection process,or weakening those edge components (that is, reducing the pixel valuesof areas that form those edges).

Through the aforementioned processing, the edge extraction unit 20obtains an image in which common edge areas have been extracted (this isreferred to as a “common edge image”). The edge extraction unit 20 thenobtains the L image common edge image, in which the coordinate locationsof the pixels in the obtained common edge image are shifted so as tomatch the coordinate locations of the pixels in the L image outputtedfrom the image input unit 102. The edge extraction unit 20 also obtainsthe R image common edge image, in which the coordinate locations of thepixels in the obtained common edge image are shifted so as to match thecoordinate locations of the pixels in the R image outputted from theimage input unit 102.

The edge extraction unit 20 then outputs the L image common edge imageto the second strength generation unit 22 of the L image correction unit21L in the second image correction unit 21 and outputs the R imagecommon edge image to the second strength generation unit 22 of the Rimage correction unit 21R in the second image correction unit 21.

FIG. 14 illustrates the configuration of the second image correctionunit 21. As shown in FIG. 14, the second image correction unit 21includes the L image correction unit 21L and the R image correction unit21R.

In FIG. 14 as well, elements that are identical to those in the imagecorrection unit 104 shown in FIG. 2 are assigned the same referencenumerals, and detailed descriptions thereof will be omitted.

The second strength generation unit 22 is inputted with the L imagecommon edge image outputted from the edge extraction unit 20. The secondstrength generation unit 22 adjusts and normalizes the levels of theaforementioned edge information in the R image common edge image or theL image common edge image inputted from the edge extraction unit 20(that is, the pixel values in the R image common edge image or the Limage common edge image). For example, the second strength generationunit 22 normalizes the inputted image signal that foams the R imagecommon edge image or the inputted image signal that forms the L imagecommon edge image to a signal in a range of [0:1].

The signal normalized by the second strength generation unit 22 of the Limage correction unit 21L is then outputted to the synthesizing unit 133as the L image strength signal K1_L.

The synthesizing unit 133 of the L image correction unit 21L synthesizesthe L image signal SS_L outputted from the smoothing unit 131 with the Limage signal IS_L outputted from the image input unit 102 based on the Limage strength signal K1_L outputted from the second strength generationunit 22 of the L image correction unit 21L.

Specifically, the synthesizing unit 133 synthesizes the L image signalIS_L and the L image signal SS_L on which the smoothing process has beenexecuted using the L image strength signal K1_L, which is a signal thathas been normalized in the range of [0:1], as an internal divisionratio, and consequently obtains the output L image signal Lout, throughthe following formula.

Note that the second strength generation unit 22 generates the L imagestrength signal K1_L by performing gain adjustment and normalization sothat the strongest edges (that is, the maximum pixel values in thecommon edge image) take on a value of 1.

Lout=(1−K1_(—) L)×IS _(—) L+K1_(—) L×SS _(—) L

In this manner, the image signal that has been synthesized by thesynthesizing unit 133 is outputted as an L output image.

Accordingly, with the three-dimensional image capturing apparatus 2000according to the present embodiment, a signal that has been smoothed inthe vicinity of the edges of the primary subject is outputted as aresult of the execution of the aforementioned processing, and thus athree-dimensional image processed by the three-dimensional imagecapturing apparatus 2000 according to the present embodiment is athree-dimensional image that has been selectively blurred only in thevicinity of the edges. As a result, the three-dimensional image obtainedby the three-dimensional image capturing apparatus 2000 according to thepresent embodiment is a three-dimensional image in which the occurrenceof the cardboard cutout effect and so on is suppressed.

Although the foregoing describes processing on the L image (processingperformed by the L image correction unit 21L), it should be noted thatthe processing performed on the R image (processing performed by the Rimage correction unit 21R) is the same.

In addition, an “end part region detection unit” is realized by the edgeextraction unit 20 and the second strength generation unit 22 of the Limage correction unit 21L when processing the L image and is realized bythe edge extraction unit 20 and the second strength generation unit 22of the R image correction unit 21R when processing the R image.

Other Embodiments

Note that the various blocks of the three-dimensional image capturingapparatus described in the aforementioned embodiments may be implementedas single individual chips, or some or all of the blocks may beimplemented as a single chip, by employing semiconductor devices such asLSIs. Note that although the term “LSI” is used here, other names, suchas IC, system LSI, super LSI, ultra LSI, and so on are used depending onthe degree of integration.

Further, the manner in which the circuit integration is achieved is notlimited to LSIs, and it is also possible to use a dedicated circuit or ageneral purpose processor. FPGAs (Field Programmable Gate Arrays) thatcan be programmed after the LSI manufacture, configurable processors inwhich the connections, settings, and so on of circuit cells within theLSIs can be reconfigured, or the like may be used as well.

Furthermore, if other technologies that improve upon or are derived fromsemiconductor technology enable integration technology to replace LSIs,then naturally it is also possible to integrate the functional blocksusing that technology. Biotechnology applications are one suchforeseeable example.

Some or all of the processing of the functional blocks of the aboveembodiments can be implemented by a program. In such a case, some or allof the processing of the functional blocks in the above embodiments arerun by a central processing unit (CPU) on a computer. A program forperforming the various processes is stored on a memory device such as ahard disk or a ROM, and is run on the ROM or read to and run on a RAM.

In addition, the various processes in the aforementioned embodiments maybe realized as hardware, or as software (this includes implementationsthrough an OS (operating system), middleware, or a predeterminedlibrary). These processes may also be implemented through processes inwhich the software and hardware run integrated with one another. It goeswithout saying that it is necessary to adjust the timing at which toexecute each process in the case where the three-dimensional imagecapturing apparatus according to the above embodiments is implementedthrough hardware. For simplicity's sake, the descriptions in the aboveembodiments have omitted the details regarding the adjustment of thetiming of the various signals that arises in the actual hardwarearchitecture.

In addition, the order of execution in the processing methods of theaforementioned embodiment are not necessarily limited to thedescriptions in the aforementioned embodiments, and the order ofexecution can be interchanged without departing from the spirit of theinvention.

A computer program that causes a computer to execute the aforementionedmethods and a computer-readable recording medium on which that programhas been recorded also fall within the scope of the present invention.Here, a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD(Blue-ray Disc), semiconductor memory, and so on can be given asexamples of such a computer-readable recording medium.

The stated computer program is not limited to a program stored on thestated recording medium, and may be transmitted via a network or thelike such as an electric communication line, a wireless or hard-wiredcommunication line, the Internet, and so on.

In addition, the aforementioned embodiments describe cases in which astereoscopic image (a left eye image and a right eye image) are obtained(captured) by two image capturing units. However, the invention is notlimited thereto, and for example, the left eye image and the right eyeimage may be obtained in an alternating manner, through time division,by a single image sensor, or the left eye image and right eye image maybe obtained by dividing a single image sensor into two image sensorsurfaces.

In addition, although the aforementioned embodiments described aconfiguration in which an R image and an L image are inputted into theimage input unit 102, the invention is not limited thereto, and forexample, an R image and an L image may be selected from N (where N is anatural number greater than or equal to 2) images of amultiple-viewpoint system, and the selected R image (signal) and L image(signal) may then be inputted into the image input unit 102.

In addition, in the three-dimensional image processing apparatus, the Rimage and the L image do not necessarily need to be obtained internally.For example, the R image and the L image may be inputted to thethree-dimensional image processing apparatus from the exterior.

Likewise, in the three-dimensional image processing apparatus, the Rdepth information and the L depth information do not necessarily need tobe obtained internally. For example, the R depth information and the Ldepth information may be inputted to the three-dimensional imageprocessing apparatus from the exterior. In this case, the depthobtainment unit 103 can be omitted from the three-dimensional imageprocessing apparatus. In other words, the three-dimensional imageprocessing apparatus may include only the image correction unit 104.

It should be noted that the specific configuration of the presentinvention is not intended to be limited to the above embodiments in anyway, and various modifications and variations can be made withoutdeparting from the essential spirit of the invention.

According to the three-dimensional image processing apparatus andthree-dimensional image processing method of the present invention, asense of three-dimensionality and thickness can be restored to a subjectand a high-quality three-dimensional image with a low sense of acardboard cutout effect can be obtained, regardless of the cause of thecardboard cutout effect. Accordingly, the present invention is useful infields related to three-dimensional images (three-dimensional video),and can be applied in such fields.

General Interpretation of Terms

In understanding the scope of the present disclosure, the term“comprising” and its derivatives, as used herein, are intended to beopen ended terms that specify the presence of the stated features,elements, components, groups, integers, and/or steps, but do not excludethe presence of other unstated features, elements, components, groups,integers and/or steps. The foregoing also applies to words havingsimilar meanings such as the terms, “including”, “having” and theirderivatives. Also, the terms “part,” “section,” “portion,” “member” or“element” when used in the singular can have the dual meaning of asingle part or a plurality of parts. Also as used herein to describe theabove embodiment(s), the following directional terms “forward”,“rearward”, “above”, “downward”, “vertical”, “horizontal”, “below” and“transverse” as well as any other similar directional terms refer tothose directions of the three-dimensional image processing apparatus,three-dimensional image processing method. Accordingly, these terms, asutilized to describe the technology disclosed herein should beinterpreted relative to the three-dimensional image processingapparatus, three-dimensional image processing method.

The term “configured” as used herein to describe a component, section,or part of a device includes hardware and/or software that isconstructed and/or programmed to carry out the desired function.

The terms of degree such as “substantially”, “about” and “approximately”as used herein mean a reasonable amount of deviation of the modifiedterm such that the end result is not significantly changed.

While only selected embodiments have been chosen to illustrate thepresent invention, it will be apparent to those skilled in the art fromthis disclosure that various changes and modifications can be madeherein without departing from the scope of the invention as defined inthe appended claims. For example, the size, shape, location ororientation of the various components can be changed as needed and/ordesired. Components that are shown directly connected or contacting eachother can have intermediate structures disposed between them. Thefunctions of one element can be performed by two, and vice versa. Thestructures and functions of one embodiment can be adopted in anotherembodiment. It is not necessary for all advantages to be present in aparticular embodiment at the same time. Every feature which is uniquefrom the prior art, alone or in combination with other features, alsoshould be considered a separate description of further inventions by theapplicants, including the structural and/or functional concepts embodiedby such feature(s). Thus, the foregoing descriptions of the embodimentsaccording to the present invention are provided for illustration only,and not for the purpose of limiting the invention as defined by theappended claims and their equivalents.

1. A three-dimensional image processing apparatus that performs an imagecorrection process, the apparatus comprising: a left eye image and aright eye image contained in a three-dimensional image obtained using adual-lens technique or a multiple-viewpoint technique; an end partregion detection unit configured to detect from the left eye image andthe right eye image at least one of: a region including an edge on aleft side of a subject contained in the left eye image and the right eyeimage, and a region including an edge on a right side of the subjectcontained in the left eye image and the right eye image, as an end partregion; and an edge correction unit configured to execute a smoothingprocess on a region in at least one end part region of the subjectdetected by the end part region detection unit.
 2. The three-dimensionalimage processing apparatus according to claim 1, wherein: the end partregion detection unit includes a depth obtainment unit configured toobtain a left eye distance image and a right eye distance image byobtaining distance information in a three-dimensional space for asubject contained in the left eye image and the right eye image; and theend part region detection unit further configured to detect the end partregion of the subject in at least one of the left eye image and theright eye image based on the distance information of the subjectobtained by the depth obtainment unit.
 3. The three-dimensional imageprocessing apparatus according to claim 1, wherein: the end part regiondetection unit includes an edge extraction unit is configured to,extract an edge of a subject contained in the left eye image and theright eye image from the left eye image and the right eye image; and theend part region detection unit is further configured to detect the endpart region of the subject, in at least one of the left eye image andthe right eye image, based on edge information of the left eye image andthe right eye image extracted by the edge extraction unit.
 4. Thethree-dimensional image processing apparatus according to claim 1,wherein: the end part region detection unit is further configured todetect the region including an edge on the left side of the subject as aleft end part region, and detect the region including an edge on theright side of the subject as a right end part region; and the edgecorrection unit is further configured to execute a smoothing process onthe left end part region in the left eye image, and execute a smoothingprocess on the right end part region in the right eye image.
 5. Thethree-dimensional image processing apparatus according to claim 1,wherein: the end part region detection unit is further configured todetect the region including an edge on the left side of the subject as aleft end part region, and detect the region including an edge on theright side of the subject as a right end part region; and the edgecorrection unit is further configured to execute: a smoothing process onthe left end part region in the left eye image at a first strength, andexecute a smoothing process on the right end part region in the left eyeimage at a second strength that is a lower strength than the firststrength; and a smoothing process on the right end part region in theright eye image at a third strength, and executes a smoothing process onthe left end part region in the right eye image at a fourth strengththat is a lower strength than the third strength.
 6. A three-dimensionalimage processing method that performs an image correction process, themethod comprising: obtaining a left eye image and a right eye imagecontained in a three-dimensional image by a dual-lens technique, or amultiple-viewpoint technique; detecting at least one end part regionfrom the left eye image and the right eye image, the at least one endpart region being at least one of: a region including an edge on a leftside of a subject contained in the left eye image and the right eyeimage, and a region including an edge on a right side of the subjectcontained in the left eye image and the right eye image, as; andsmoothing a region in the at least one end part region of the subject.